This page documents the progress and status of Benthic Rover data management activities, as of the end of July, 2009.

Overview: Status of Benthic Rover Data Mgt Tasks

Right now Rover data is being copied (manually?) to shore. The intent is to have it all logged to SSDS. The proposal for doing so was previously circulated to the Rover team, and has a lot of detail that may be of interest, though some information is out of date.

The Rover data is stored in the project share for the Rover (900502.BenthicRover), under Rover.Deployment. This is not an ideal place for data; the permissions have to be set correctly to avoid accidentally losing the data, and I believe it can not be exposed for easy access via the web. The project may wish to consider moving it to a more accessible share, similar to the BIAUV share.

There are several tasks for Rover data management. The tasks and their status are as follows.

Convert most recent Rover data to new format

Some time ago we designed a new data format (changing both organization of files, and content for the data files). Past Rover data of value (through last October, the last pre-MARS mission) were converted to the new format. This data is stored under Rover.Documents/Project.Data/ReprocessedData/RoverData_2007-2008_NewFormat.

The Rover began collecting data again several weeks ago, when it was attached to the MARS node. That data is being uploaded to the Rover Project Folder on Tornado. It appears the current release of the Rover software writes data in this format. (nice, Rich!) See Converting Rover Data Formats below for more information.

Logging Rover data in SSDS

No Rover data is being logged to SSDS; this task is in progress as described below.

Originally the notion was to log Rover data in real time. Since the Rover will only be on MARS for a brief period, we no longer intend to do this, but will log all data retroactively instead.

Logging Rover images to SSDS

This task has not started. The approaches being considered are described below.

Converting Rover Data Formats

This task established a 'normalized' data file format, in which each mission (e.g., a cruise), deployment (when Rover is physically deployed off a ship, or spends a period of time doing a science or engineering project), plan (script or similar set of commands), or activity (something producing data from a particular device) had its own folder. Folder names indicate the type of folder, the sequence of the folder among others at the same level, and whatever identifier the original user wanted for that mission, deployment, plan, or activity. This provides an order and hierarchy for the data, and puts different kinds of data into different folders.

Each data file also has a standard format in the normalized scheme, making fields like timestamp the same for all Rover files.

The organization and renaming of files from the old format to the new format was performed manually, and only for those missions that appeared to have any data of interest that could be converted. The reformatting of records was done automatically via the formatDataFiles.pl script (checked in to svn:rover/trunk/scripts/perl/). This script leaves filenames in an intermediate state; the user must then run undoFormatFiles.pl (in the same directory; has a terribly misleading name) to either finalize or revoke the changes. Not ideal, but I don't know that you'll have to run this code ever again.

It appears the MARS Roverdeployment is writing missions largely in the new format (yay!). Only the system log and optode file formats could be confirmed; currents data was not evident yet. Metadata about device IDs is missing from the first line of the data files; either this will need to be corrected, or the processing software will need to know what to use for a device ID when none is present.

There are some minor differences between the proposed directory layout and the new layout (most of these can be addressed trivially), and some new files are present that may require additional code to submit.

Submitting Rover Data to SSDS

The strategy for submitting Rover data was to submit it record by record to the SSDS system, associating the records with the corresponding device IDs using the XML at the top of the file. Metadata will be collected from the folder names, and submitted separately to SSDS. The hierarchy of the folders can be preserved in SSDS by making each folder a deployment in SSDS, thus allowing similar navigation within SSDS.

(We considered submitting all the original files of Rover data to SSDS, the way the AUVs do. I considered the files more of a transport mechanism, and also wanted to support real-time submission at some point, so this is not the current mechanism.)

So far we have successfully validated submitting individual test records to SSDS (not Rover test records, but the difference should be trivial). And I have a Perl script checked in, roverParse.pl, that can either submit records to SSDS, or log them to a file; this appeared to be working (logging to a file) when run against the reformatted data. All that would be required to submit the data to SSDS is setting a flag to submit data into the SSDS server, instead of logging it (recommend running against a test server first though, especially if you wanted to run it on the new MARS data, as it hasn't had much testing).

I have some notional changes on where to fit metadata management into that script. Some Perl was written to parse folder names, but it is in a very preliminary state. And I started to verify the metadata submission process using SSDS testing utilities, but there were challenges I haven't overcome. All this work was continuing up until my departure.

The next steps will be to (a) try parsing and submitting all the metadata information; (b) try submitting all the data records (hmm, might be better/faster to do this first?) (c) connect the data to the metadata, or vice versa. Since data goes into SSDS buckets according to the device that generated it, it will be important for the processing scripts to either know, or have a way to determine, those device IDs. (Paul McGill keeps a spreadsheet as current as he can that contains this information; make sure he updates the one that I added a sheet to, as it has a more thorough description of device IDs on all the deployments.)

The data records were most important data elements, but more data files appear to be in the data sets now (not sure which of the additional files are generated on the Rover). A decision will have to be made about which of these should be submitted to the SSDS. The team may also prefer to keep processed data separate from the original data, to avoid confusion.

Submitting Rover Images to SSDS

The Rover collects images from multiple cameras. Some of these are JPEG images, and some are RAW (Bayer) images. The intent was to either submit the images to SSDS directly, or store them in an accessible place and index them in SSDS. Unlike the BIAUV Image Processing Strategy situation, there are relatively few images for the Rover, so storing them directly in SSDS is not out of the question.

The exact technique for storing images was still being investigated, and the metadata for the images needs consideration.

Conversion of images

The RAW images must be converted to JPEG or other suitable format. Whether this is done before or after storing the images is at the discretion of the program, given the disk space required/available. Although no Unix script existed to do this when last investigated (in 2008), one may be available now.

Metadata for images

The image metadata in the raw images is entirely non-existent. There is not even a timestamp. Fortunately timestamps are maintained in the name of the image, but this is a very weak metadata system for such a critical piece of information.

The plan and recommendation is to follow similar image metadata post-processing as is performed for the Benthic Imaging AUV images (BIAUV Image Processing Strategy). Much of the same code could be reused, but the metadata will have to come from different places or analysis. (The fastest way to do this may be to generate netCDF files for the Rover data sets, as this could probably be done readily. Then much of the BIAUV processing could apply more directly.) Rover position should be assumed at first to be the same as the MARS node, with a large error bar of course.

The most fundamental and important modification to add metadata for images will be the timestamp, as all knowledge of the image depends on correct timestamps, and putting that metadata in the name is very weak, as noted previously.

Image storing technique

As discussed in the BIAUV Image Processing Strategy, it is not clear whether images should be stored directly in SSDS at all. Kevin Gomes and I were interested in trying this out to see how well it could work, snd I did a bit of work on Rover with it. It might be advantageous to use the same method as the BIAUV; the team may want to read that document before deciding on an approach. The detailed organization of the Rover data products, where images are in many different sub-sub-directories, may be a factor.

The first attempt to store images was via hex-encoded URLs (HTTP GET), but this failed due to the length (MBs!) of the URL.

A second suggestion, not yet tried, was submitting images via HTTP POST.

It is also possible to directly submit images via a Java interface. This is awkward with the Perl scripts, but a Java tool might do so readily.

Finally, it is possible to just store the images on an appropriately accessible share, and index them in the SSDS metadata.

XML and XSLT Files

Rich Henthorn and John Graybeal spent some time figuring out how to embed descriptions of the Rover deployed instrumentation in (a) an on-board description file or directory, and (b) the data logs of a mission. The notion we came up with was to document all the devices in XML files that are on board, and create a single master XML configuration file that points to the appropriate XML files (using XPath and XPointer). In addition, the first line of data files could contain a minimal XML segment containing the device IDs; this would be written by the device driver, and is needed for post-processing to submit the data records into appropriate device bins.

I'm not sure how much of those concepts are in the Rover mission code, nor how much will be added later. The XML files have been written and are checked in at svn:rover/trunk/xml and the RoverConfiguration subdirectory.

The Rover configurations can be described in a nice format using the XML files and nice XSLT transforms to convert the XML information into web pages. Three XSLT files, to transform the XML into descriptions or checklists of the Rover configuration, can be found at svn:rover/trunk/scripts/xsl. Example outputs are also there. See the README file for details.

These transformations can be run on the Rover itself using (for example) xalan or similar UNIX-generic XSL transformation software, but this environment has not been set up on the Rover, to my knowledge. (Note Oxygen's xsltproc does not correctly handle the XPath/XPointer, and so will not work; you must set the XSLT processor as part of the Transformation configuration settings.)